I'm trying to download some spacenet data (23 GB) via the following command:
aws s3api get-object --bucket spacenet-dataset --key AOI_2_Vegas/AOI_2_Vegas_Train.tar.gz --request-payer requester AOI_2_Vegas_Train.tar.gz
My default region is us-east-1 and I know my keys are correct because by account is being charged.
I ran this at my home (30 mbps) and it timed out after about 12 GB.
I ran this at my work (200 mbps) and it timed out after about 16 GB.
Here is the Error:
HTTPSConnectionPool(host='spacenet-dataset.s3.amazonaws.com', port=443): Read timed out.
Is there a way to resume an "aws s3api get-object" request that has timed out?
I would maybe try to work with the aws s3 cp command. This command is more high-level and can automatically handle multipart transfer.
This command can also take few options that can be added. see http://docs.aws.amazon.com/cli/latest/topic/s3-config.html
specially at your work you can increase max_concurrent_requests and multipart_threshold so for example, configure as follow
$ aws configure set default.s3.max_concurrent_requests 25
$ aws configure set default.s3.multipart_threshold 128MB
$ aws configure set default.s3.multipart_chunksize 32MB
$ aws configure set default.s3.use_accelerate_endpoint true
and run the copy command.
It is possible to download a specified byte range of an object. This can be done by using the range flag as follows.
aws s3api get-object --bucket spacenet-dataset --key AOI_2_Vegas/AOI_2_Vegas_Train.tar.gz --request-payer requester --range bytes=0-99999 file_1.tar.gz
The command above downloads the first 100,000 bytes of the 23 GB file and writes it to file_1.tar.gz.
Once all file_*.tar.gz broken in byte ranges are downloaded they would have to be concatenated in the correct order. It can be done using the *nix cat command as follows.
cat file_1.tar.gz >> AOI_2_Vegas_Train_all.tar.gz
cat file_2.tar.gz >> AOI_2_Vegas_Train_all.tar.gz
Check this out for more details.